<html lang="en-US">

<head>
    
<title>爬取豆瓣电影 Top250 - 花落雨忧</title>

<meta property="og:title" content="爬取豆瓣电影 Top250 - 花落雨忧">



    



    
    <meta property="description" content="爬虫是标配了，看数据那一刻很有趣。第一个就从最最最简单最基础的爬虫开始写起吧！
项目地址：https://github.com/go-crawler/douban-movie
[&amp;hellip;] 我们的目标站点是 豆瓣电影 Top250，估计大家都很眼熟了
本次爬取 8 个字段，用于简单的概括分析。具体的字段如下：
简单的分析一下目标源
[&amp;hellip;] 由于量不大， &amp;hellip;">
    <meta property="og:description" content="爬虫是标配了，看数据那一刻很有趣。第一个就从最最最简单最基础的爬虫开始写起吧！
项目地址：https://github.com/go-crawler/douban-movie
[&amp;hellip;] 我们的目标站点是 豆瓣电影 Top250，估计大家都很眼熟了
本次爬取 8 个字段，用于简单的概括分析。具体的字段如下：
简单的分析一下目标源
[&amp;hellip;] 由于量不大， &amp;hellip;">
    






<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width,initial-scale=1">

<link rel="shortcut icon" href="https://www.lican.asia/logo/logo.png" type="image/x-icon" />



<link rel="stylesheet" href="/css/style.min.css" />

<link rel="stylesheet" href="/css/reset.min.css" />




<script src="https://www.lican.asia/js/highlight.min.js"></script>

<script>
hljs.configure({ ignoreUnescapedHTML: true })
hljs.highlightAll();
</script>


<script src="https://www.lican.asia/js/jquery.min.js"></script>




<link href="https://www.lican.asia/css/hugo-code.min.css" rel="stylesheet" />



    <style>
        .post-content img {
            max-width: 400px;
        }
    </style>
</head>

<body id="period" class="home blog">
    <a class="skip-content" href="#main">Press "Enter" to skip to content</a>
    <div id="overflow-container" class="overflow-container">
        <header class="site-header" id="site-header" role="banner">
    <div class="max-width">
        <div id="title-container" class="title-container">
            <div id="site-title" class="site-title"><a href="/">花落雨忧</a></div>
            <p class="tagline">思所及 力所行 方为真.</p>
        </div>
        
        <div id="menu-primary-container" class="menu-primary-container">
            <div class="icon-container">
            </div>
            <div id="menu-primary" class="menu-container menu-primary" role="navigation">
                <nav class="menu">
                    <ul id="menu-primary-items" class="menu-primary-items">
                        
                        
                        <li id="menu-item-0"
                            class="menu-item menu-item-type-custom menu-item-object-custom ">
                            <a href="/posts" aria-current="page" tabindex="0">首页</a></li>
                        
                        <li id="menu-item-1"
                            class="menu-item menu-item-type-custom menu-item-object-custom ">
                            <a href="/tech/" aria-current="page" tabindex="1">技术文档</a></li>
                        
                        <li id="menu-item-2"
                            class="menu-item menu-item-type-custom menu-item-object-custom ">
                            <a href="/article/" aria-current="page" tabindex="2">文章</a></li>
                        
                        <li id="menu-item-3"
                            class="menu-item menu-item-type-custom menu-item-object-custom ">
                            <a href="/project/" aria-current="page" tabindex="3">项目</a></li>
                        
                        <li id="menu-item-4"
                            class="menu-item menu-item-type-custom menu-item-object-custom ">
                            <a href="/about/" aria-current="page" tabindex="4">关于</a></li>
                        
                    </ul>
                </nav>
            </div>
        </div>
    </div>
</header>

        <div id="primary-container" class="primary-container">
            <div class="max-width">
                <section id="main" class="main" role="main">
                    <div id="loop-container" class="loop-container">
                        <div
                            class="post type-post status-publish format-standard hentry entry">
                            <article>
                                <div class="post-container">
                                    <div class="post-header">
                                        <h2 class="post-title">
                                            <a href="/posts/posts/go/crawler/2018-03-21-douban-top250/">爬取豆瓣电影 Top250</a>
                                        </h2>
                                        
                                        <div class="post-byline">Published on
                                            <a class="date" href="javascript:;">2018/03/21</a>
                                            
                                            
                                            
                                            By 
                                            <a class="author" href="javascript:;">煎鱼</a></div>
                                            
                                        
                                    </div>
                                    <div class="post-content">
                                        <p>爬虫是标配了，看数据那一刻很有趣。第一个就从最最最简单最基础的爬虫开始写起吧！</p>
<p>项目地址：https://github.com/go-crawler/douban-movie</p>
<h2 id="目标">目标</h2>
<p>我们的目标站点是 <a href="https://movie.douban.com/top250">豆瓣电影 Top250</a>，估计大家都很眼熟了</p>
<p>本次爬取 8 个字段，用于简单的概括分析。具体的字段如下：</p>
<p><img src="https://i.loli.net/2018/03/20/5ab11596b8810.png" alt="image"></p>
<p>简单的分析一下目标源</p>
<ul>
<li>一页共 25 条</li>
<li>含分页（共 10 页）且分页规则是正常的</li>
<li>每一项的数据字段排序都是规则且不变</li>
</ul>
<h2 id="开始">开始</h2>
<p>由于量不大，我们的爬取步骤如下</p>
<ul>
<li>分析页面，获取所有的分页</li>
<li>分析页面，循环爬取所有页面的电影信息</li>
<li>爬取的电影信息入库</li>
</ul>
<h3 id="安装">安装</h3>
<pre tabindex="0"><code>$ go get -u github.com/PuerkitoBio/goquery
</code></pre><h3 id="运行">运行</h3>
<pre tabindex="0"><code>$ go run main.go
</code></pre><h3 id="代码片段">代码片段</h3>
<h4 id="1获取所有分页">1、获取所有分页</h4>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">ParsePages</span>(<span style="color:#a6e22e">doc</span> <span style="color:#f92672">*</span><span style="color:#a6e22e">goquery</span>.<span style="color:#a6e22e">Document</span>) (<span style="color:#a6e22e">pages</span> []<span style="color:#a6e22e">Page</span>) {
</span></span><span style="display:flex;"><span>	<span style="color:#a6e22e">pages</span> = append(<span style="color:#a6e22e">pages</span>, <span style="color:#a6e22e">Page</span>{<span style="color:#a6e22e">Page</span>: <span style="color:#ae81ff">1</span>, <span style="color:#a6e22e">Url</span>: <span style="color:#e6db74">&#34;&#34;</span>})
</span></span><span style="display:flex;"><span>	<span style="color:#a6e22e">doc</span>.<span style="color:#a6e22e">Find</span>(<span style="color:#e6db74">&#34;#content &gt; div &gt; div.article &gt; div.paginator &gt; a&#34;</span>).<span style="color:#a6e22e">Each</span>(<span style="color:#66d9ef">func</span>(<span style="color:#a6e22e">i</span> <span style="color:#66d9ef">int</span>, <span style="color:#a6e22e">s</span> <span style="color:#f92672">*</span><span style="color:#a6e22e">goquery</span>.<span style="color:#a6e22e">Selection</span>) {
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">page</span>, <span style="color:#a6e22e">_</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">strconv</span>.<span style="color:#a6e22e">Atoi</span>(<span style="color:#a6e22e">s</span>.<span style="color:#a6e22e">Text</span>())
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">url</span>, <span style="color:#a6e22e">_</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">s</span>.<span style="color:#a6e22e">Attr</span>(<span style="color:#e6db74">&#34;href&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">pages</span> = append(<span style="color:#a6e22e">pages</span>, <span style="color:#a6e22e">Page</span>{
</span></span><span style="display:flex;"><span>			<span style="color:#a6e22e">Page</span>: <span style="color:#a6e22e">page</span>,
</span></span><span style="display:flex;"><span>			<span style="color:#a6e22e">Url</span>:  <span style="color:#a6e22e">url</span>,
</span></span><span style="display:flex;"><span>		})
</span></span><span style="display:flex;"><span>	})
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>	<span style="color:#66d9ef">return</span> <span style="color:#a6e22e">pages</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><h4 id="2分析豆瓣电影信息">2、分析豆瓣电影信息</h4>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">ParseMovies</span>(<span style="color:#a6e22e">doc</span> <span style="color:#f92672">*</span><span style="color:#a6e22e">goquery</span>.<span style="color:#a6e22e">Document</span>) (<span style="color:#a6e22e">movies</span> []<span style="color:#a6e22e">Movie</span>) {
</span></span><span style="display:flex;"><span>	<span style="color:#a6e22e">doc</span>.<span style="color:#a6e22e">Find</span>(<span style="color:#e6db74">&#34;#content &gt; div &gt; div.article &gt; ol &gt; li&#34;</span>).<span style="color:#a6e22e">Each</span>(<span style="color:#66d9ef">func</span>(<span style="color:#a6e22e">i</span> <span style="color:#66d9ef">int</span>, <span style="color:#a6e22e">s</span> <span style="color:#f92672">*</span><span style="color:#a6e22e">goquery</span>.<span style="color:#a6e22e">Selection</span>) {
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">title</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">s</span>.<span style="color:#a6e22e">Find</span>(<span style="color:#e6db74">&#34;.hd a span&#34;</span>).<span style="color:#a6e22e">Eq</span>(<span style="color:#ae81ff">0</span>).<span style="color:#a6e22e">Text</span>()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>		<span style="color:#f92672">...</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">movieDesc</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">strings</span>.<span style="color:#a6e22e">Split</span>(<span style="color:#a6e22e">DescInfo</span>[<span style="color:#ae81ff">1</span>], <span style="color:#e6db74">&#34;/&#34;</span>)
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">year</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">strings</span>.<span style="color:#a6e22e">TrimSpace</span>(<span style="color:#a6e22e">movieDesc</span>[<span style="color:#ae81ff">0</span>])
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">area</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">strings</span>.<span style="color:#a6e22e">TrimSpace</span>(<span style="color:#a6e22e">movieDesc</span>[<span style="color:#ae81ff">1</span>])
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">tag</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">strings</span>.<span style="color:#a6e22e">TrimSpace</span>(<span style="color:#a6e22e">movieDesc</span>[<span style="color:#ae81ff">2</span>])
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">star</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">s</span>.<span style="color:#a6e22e">Find</span>(<span style="color:#e6db74">&#34;.bd .star .rating_num&#34;</span>).<span style="color:#a6e22e">Text</span>()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">comment</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">strings</span>.<span style="color:#a6e22e">TrimSpace</span>(<span style="color:#a6e22e">s</span>.<span style="color:#a6e22e">Find</span>(<span style="color:#e6db74">&#34;.bd .star span&#34;</span>).<span style="color:#a6e22e">Eq</span>(<span style="color:#ae81ff">3</span>).<span style="color:#a6e22e">Text</span>())
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">compile</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">regexp</span>.<span style="color:#a6e22e">MustCompile</span>(<span style="color:#e6db74">&#34;[0-9]&#34;</span>)
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">comment</span> = <span style="color:#a6e22e">strings</span>.<span style="color:#a6e22e">Join</span>(<span style="color:#a6e22e">compile</span>.<span style="color:#a6e22e">FindAllString</span>(<span style="color:#a6e22e">comment</span>, <span style="color:#f92672">-</span><span style="color:#ae81ff">1</span>), <span style="color:#e6db74">&#34;&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">quote</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">s</span>.<span style="color:#a6e22e">Find</span>(<span style="color:#e6db74">&#34;.quote .inq&#34;</span>).<span style="color:#a6e22e">Text</span>()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>		<span style="color:#f92672">...</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">log</span>.<span style="color:#a6e22e">Printf</span>(<span style="color:#e6db74">&#34;i: %d, movie: %v&#34;</span>, <span style="color:#a6e22e">i</span>, <span style="color:#a6e22e">movie</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>		<span style="color:#a6e22e">movies</span> = append(<span style="color:#a6e22e">movies</span>, <span style="color:#a6e22e">movie</span>)
</span></span><span style="display:flex;"><span>	})
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>	<span style="color:#66d9ef">return</span> <span style="color:#a6e22e">movies</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><h3 id="数据">数据</h3>
<p><img src="https://i.loli.net/2018/03/21/5ab1309594741.png" alt="image"></p>
<p><img src="https://i.loli.net/2018/03/21/5ab131ca582f8.png" alt="image"></p>
<p><img src="https://i.loli.net/2018/03/21/5ab130d3a00d9.png" alt="image"></p>
<p>看到这些数据，你有什么想法呢，真是好奇 :=)</p>


                                        
                                        
                                        
                                        <div class="rp4wp-related-posts">
                                            <h3>相关文章</h3>
                                            <ul>
                                                
                                                <li>
                                                    <div class="rp4wp-related-post-content">
                                                        <a
                                                            href="https://www.lican.asia/posts/posts/go/gin/2018-03-18-swagger/">「连载八」为它加上Swagger</a>
                                                    </div>
                                                </li>
                                                
                                                <li>
                                                    <div class="rp4wp-related-post-content">
                                                        <a
                                                            href="https://www.lican.asia/posts/posts/go/gin/2018-03-15-reload-http/">「连载七」优雅的重启服务</a>
                                                    </div>
                                                </li>
                                                
                                                <li>
                                                    <div class="rp4wp-related-post-content">
                                                        <a
                                                            href="https://www.lican.asia/posts/posts/go/talk/2018-03-13-golang-relatively-path/">聊一聊，Go 的相对路径问题</a>
                                                    </div>
                                                </li>
                                                
                                                <li>
                                                    <div class="rp4wp-related-post-content">
                                                        <a
                                                            href="https://www.lican.asia/posts/posts/go/grpc-gateway/2018-03-04-swagger/">「连载三」Swagger了解一下</a>
                                                    </div>
                                                </li>
                                                
                                                <li>
                                                    <div class="rp4wp-related-post-content">
                                                        <a
                                                            href="https://www.lican.asia/posts/posts/go/grpc-gateway/2018-02-27-hello-world/">「连载二」Hello World</a>
                                                    </div>
                                                </li>
                                                
                                            </ul>
                                        </div>
                                        
                                        
                                    </div>

                                    
                                    
                                    

                                    
                                    <div class="post-meta">
                                        
                                        
                                        <div class="post-tags">
                                            <ul>
                                            
                                            <li>
                                                <a href="/tags/go" title="View all posts tagged match">go</a>
                                            </li>
                                            
                                            <li>
                                                <a href="/tags/%e6%95%b0%e6%8d%ae%e5%88%86%e6%9e%90" title="View all posts tagged match">数据分析</a>
                                            </li>
                                            
                                            </ul>
                                        </div>
                                        
                                        
                                        <nav class="further-reading">
                                            
                                            <div class="previous">
                                                <span>&lt;&lt; Prev</span>
                                                <a href="https://www.lican.asia/posts/posts/go/gin/2018-03-18-swagger/"
                                                    rel="prev">「连载八」为它加上Swagger</a> </div>
                                            
                                            
                                            <div class="next">
                                                <span>Next >></span>
                                                <a href="https://www.lican.asia/posts/posts/go/gin/2018-03-24-golang-docker/">「连载九」将Golang应用部署到Docker</a> 
                                            </div>
                                            
                                        </nav>
                                    </div>
                                    
                                    

                                    
                                    
                                    

                                </div>
                            </article>
                        </div>
                    </div>
                </section>
                <aside class="sidebar sidebar-primary" id="sidebar-primary" role="complementary">
    <h1 class="screen-reader-text">Sidebar</h1>
    

    
    
    <section id="text-2" class="widget widget_text">
        <div class="textwidget">
            
            <div id="profile">
                <div id="profile_picture"><img src="https://www.lican.asia/logo/logo.png"></div>
                <div id="profile_intro">
                    <p><span class="name">Lican</span></p>
                    <p class="intro">全栈开发者，爱好造轮子。</p>
                </div>
            </div>
            
            <p>
                <script type="text/javascript">
                    (function ($) {
                        $(document).ready(function () {
                            var menuPrimaryContainer = $('#menu-primary-container');
                            var profile = $('#text-2');
                            $('#toggle-navigation').click(function () {
                                if (menuPrimaryContainer.hasClass('open')) {
                                    profile.removeClass('open');
                                } else {
                                    profile.addClass('open');
                                }
                            });
                        });
                    })(jQuery);
                </script>
            </p>
        </div>
    </section>
    
    
    
    
    
    <section id="text-5" class="widget widget_text">
        <h2 class="widget-title">开源项目</h2>
        <div class="textwidget">
            <div id="projects" style="line-height: 22px;">
                
                <a href="https://github.com/idoubi/gonews"
                    target="_blank">gonews</a>: &nbsp;Daily news for golang<br>
                
                <a href="https://github.com/idoubi/sql2struct"
                    target="_blank">sql2struct</a>: &nbsp;Generate go struct according to SQL<br>
                
                <a href="https://github.com/idoubi/goz"
                    target="_blank">goz</a>: &nbsp;Request library used in golang<br>
                
        </div>
    </section>
    
    

    
    
    
    
    <section id="qrcode" class="widget widget_media_image">
        <h2 class="widget-title">微信公众号</h2>
        <img width="258" height="258"
            src="https://www.lican.asia/wechat/lican.png"
            class="image wp-image-5514  attachment-full size-full" alt=""
            style="max-width: 100%; height: auto;"
            sizes="(max-width: 258px) 100vw, 258px">
    </section>
    
    

    
    
    

    
    
    

    
    
    
    
</aside>
            </div>
        </div>

        <footer id="site-footer" class="site-footer" role="contentinfo">
    <div class="max-width">
    </div>
    <div class="footer">
        <div id="footercontent">
            © lican.asia All rights reserved<br/>
            Built with Hugo Theme <a href="https://github.com/idoubi/hugo-theme-period" target="_blank">Period</a>
        </div>
    </div>
</footer>

<script>
    var _hmt = _hmt || [];
    (function() {
      var hm = document.createElement("script");
      hm.src = "https://hm.baidu.com/hm.js?e8351b6d4626d5881d439ea1f6184baa";
      var s = document.getElementsByTagName("script")[0]; 
      s.parentNode.insertBefore(hm, s);
    })();
</script>
    
    
    </div>
    
</body>

</html>